Systems and methods for wide-angle LiDAR using non-uniform magnification optics

ABSTRACT

Methods and systems for combining information from a first image captured of a scene via a first sensor and information from a second image captured of the scene via a second sensor wherein the first image and second image have at least one common field of view (FoV) and wherein the first image comprises pixels that are distributed according to a non-linear image point distribution function. The first image is corrected, before combining, based on said non-linear distribution function.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of and claims the benefitof priority under 35 U.S.C. § 120 to U.S. application Ser. No.17/382,144, filed on Jul. 21, 2021, which claims the benefit of U.S.Provisional Application No. 63/054,634, filed on Jul. 21, 2020, thecontents of which are hereby incorporated by reference.

FIELD

This disclosure relates generally to LiDAR systems and methods ofoperation and, in particular, to a method for operating a LiDAR systemacross a wide-angle field-of-view.

BACKGROUND

LiDAR systems can be used in various applications, such as in vehicles,portable computer devices (e.g., smartphones, laptops, tablets) andaugmented/virtual reality devices/systems, in order to image a field ofview and locate objects within the field of view. A LiDAR system directslight outward over a range of angles and receives reflections of thelight from objects. Many current LiDAR systems use a mechanical-scanningdevice, such as a gimbal or spinning disks or polygons in order todisperse outgoing light beams. However, such mechanical-scanning devicesoften come with resolution issues, maintenance issues, assembly issuesand/or temperature dependence issues.

For these and other reasons, there is a need to improvemanufacturability, performance and use of LiDAR systems in aspects suchas range, resolution, field-of-view, and physical and environmentalrobustness.

BRIEF DESCRIPTION OF DRAWINGS

A detailed description of embodiments is provided below, by way ofexample only, with reference to drawings accompanying this description,in which:

FIG. 1 shows an example of a LiDAR system transmitting an opticalimpulse into a field of view and determining range of objects based ontime of flight of echoes reflected back from the objects within thefield of view.

FIG. 2 shows examples of basic LiDAR system components for detection andranging.

FIG. 3 shows an example of an autonomous driving vehicle configured as amulti-person shuttle with a conventional mechanical-scanning LiDARsensor mounted on the top of the shuttle near the front of the shuttle.

FIG. 4 shows examples of external objects having features above groundlevel that can pose detection problems for conventional LiDAR systems.

FIG. 5 shows an example of a potential use for a wide-angle LiDAR systemfor turning assistance;

FIG. 6 shows another example of a potential use for a wide-angle LiDARsystem for blind spot coverage;

FIGS. 7A, 7B and 7C show top, side and front views, respectively, of ashuttle vehicle having a high side-mounted wide-angle LiDAR system witha Field of View that extends in a vertical direction substantially 90°from the horizon to the ground and in a horizontal directionsubstantially 180° from the rear of the shuttle to the front of theshuttle.

FIG. 8 shows an example of a uniform vertical angular distributionextending over substantially 90° from the horizon to the ground.

FIGS. 9A and 9B show example plots of simulated LiDAR pixel data for aLiDAR system having the uniform vertical angular distribution of FIG. 8with a 0.5 m×1.8 m target at distances of 15 m and 5 m, respectively.

FIG. 10 shows an example of a non-uniform vertical angular distributionextending over substantially 90° from the horizon to the ground.

FIG. 11 shows an example of a segmented FoV having areas of non-uniformvertical resolution based on the non-uniform vertical angulardistribution of FIG. 10 over substantially 90° from the horizon to theground and uniform horizontal resolution over substantially 180° in thehorizontal direction.

FIGS. 12A, 12B and 12C show example plots of simulated LiDAR pixel datafora LiDAR system having the non-uniform vertical angular distributionof FIGS. 10 and 11 with a 0.5 m×1.8 m target at distances of 5 m, 15 mand 50 m, respectively.

FIG. 13 shows an example of magnification optics used to amplify theangular distribution of an optical emitter module in accordance with anembodiment of the present disclosure.

FIG. 14 shows an example of a digital beam steering componentimplemented using a liquid crystal polarization grating (LCPG) inaccordance with an embodiment of the present disclosure.

FIG. 15 shows an example of two dimensional (2D) beam steering anglesthat are possible using the LCPG beam steering element of FIG. 14 .

FIG. 16 shows an example plot of transmission and reception efficienciesvs. steering angle for the LCPG beam steering element of FIG. 14 .

FIG. 17 shows tables of examples of non-uniform steering angleconfigurations and corresponding geometric distances at heights of 2.5 mand 3.5 m for the LiDAR system of FIG. 7C.

FIG. 18 shows a top down view of an example of a LiDAR system withnon-uniform magnification optics that may be used to implement thesegmented FoV with non-uniform vertical resolution and uniformhorizontal resolution of FIG. 11 .

FIG. 19 shows a side on view of the LiDAR system of FIG. 18 showing thenon-uniform vertical steering angles resulting from the non-uniformmagnification optics.

FIG. 20 shows the firing sequence for the light source and theconfiguration of the sensor unit of the LiDAR system of FIGS. 18 and 19.

FIG. 21 shows an example of an accumulation strategy for the segments ofthe segmented FoV represented by the steering angle configurations ofthe LCPG of the LiDAR system of FIGS. 18 and 19 .

FIG. 22 shows an example of another accumulation strategy for thesegments of the segmented FoV with unequal accumulations along thehorizontal direction.

FIG. 23 shows a top down view of an example of a LiDAR system withnon-uniform magnification optics in which the emitter module and thereception module have separate magnification optics.

FIG. 24 shows a top down view of an example of a LiDAR system withnon-uniform magnification optics in which the optical emitter andreception paths are co-axial.

FIG. 25 shows a top down view of an example of a LiDAR system withnon-uniform magnification optics and a beam steering device implementedby a MEMS device.

FIG. 26 shows a top down view of an example of a LiDAR system withnon-uniform magnification optics and a beam steering device implementedby an LCPG and a MEMS device.

FIG. 27 shows a top down view of an example of a LiDAR system withnon-uniform magnification optics and a FLASH structure for optical beamdispersal.

FIG. 28 shows another example of a LiDAR system with non-uniformmagnification optics according to an embodiment of the presentdisclosure.

FIG. 29 shows a flowchart of a method according to another embodiment ofthe present disclosure.

FIG. 30 shows another example of a LiDAR system with non-uniformmagnification optics according to an embodiment of the presentdisclosure.

FIG. 31 shows another example of non-linearity of magnification opticsimplemented by a panoramic objective lens according to the presentinvention.

FIG. 32 shows an example of an apparatus that includes a LiDAR systemand an image system, at least one of which has non-uniform magnificationoptics according to an embodiment of the present disclosure.

FIG. 33 shows a flowchart of a method for merging LiDAR data with imagedata according to another embodiment of the present disclosure.

FIG. 34 shows an example of an apparatus that includes a LiDAR systemand an image system that share common non-uniform magnification opticsaccording to an embodiment of the present disclosure.

It is to be expressly understood that the description and drawings areonly for purposes of illustrating certain embodiments and are an aid forunderstanding. They are not intended to be and should not be limiting.

DETAILED DESCRIPTION OF EMBODIMENTS

LiDAR Systems

Radiation with wavelength in the optical region of the electromagneticspectrum i.e., from the ultraviolet up to the infrared, can interactwith matter in various states through mechanisms such as opticalabsorption and scattering. Early after the advent of the first lasers,it was recognized that these novel sources of coherent optical radiationcould be used for sensing solid objects, particulate matter, aerosols,and even molecular species located at long distances. Remote sensingapplications emerged owing to some distinctive features of lasersources. For example, several types of laser sources emit optical pulsescarrying high energy that can propagate in the atmosphere in the form ofa slowly-diverging optical beam. Similarly to the radio and microwaveradiation sources used in common radar instruments, systems that employlight sources for remote sensing applications are generally known asLiDAR systems, or simply LiDARs, which is the acronym for LightDetection And Ranging.

LiDAR works much like radar, emitting optical light pulses (e.g.,infrared light pulses) of short duration, typically in the ns(nanosecond, 1 ns=10⁻⁹ s) range, either in single-shot regime or in theform of a pulse train of limited duration, instead of radio waves andmeasuring how long they take to come back after hitting nearby objects.This is shown conceptually in FIG. 1 , which shows a Lidar system 10transmitting an output laser pulse and receiving echoes from twoobjects. The time between the output laser pulse and the reflectedpulses allows the LiDAR sensor to calculate the distance to each objectprecisely, based on the speed of light. For example, the range R of anobject may be deduced from the measured full round-trip time T of theoptical pulse using the simple relation:

${R = \frac{cT}{2n}},$

where c is the speed of light in vacuum, which scales to roughly 3×10⁸m/s, and n denotes the refractive index of the medium in which theoptical pulse propagates. Methods for optical ranging are not limited tothe pulsed TOF technique. Methods such as optical triangulation,interferometric phase-shift range finding, and frequency-modulatedcontinuous-wave (FMCW) range finding, just to name of few, exist aswell. The review paper of M.-C. Amann et al. (“Laser ranging: a criticalreview of usual techniques for distance measurement”, OpticalEngineering vol. 40, pp. 10-19, January 2001) discusses these techniquesin greater details.

LiDAR systems may be capable of capturing millions of such precisedistance measurement points each second, from which a 3D matrix of itsenvironment can be produced. Information on objects' position, shape,and behavior can be obtained from this comprehensive mapping of theenvironment, as shown in the example mapping shown in FIG. 1 .

General Overview of a LiDAR System

The various embodiments of the present disclosure described below areintended for implementation in LiDAR system with non-uniformmagnification optics. Some of the basic elements of a LiDAR system 10may be better appreciated by referring to the schematic block diagramdepicted in FIG. 2 . The LiDAR system 10 comprises an optical emittermodule 12 for emission of a train of optical pulses having predeterminedcharacteristics, and an optical receiver module 14 for the capture andpre-processing of the return signal waveforms. For example, the signalwaveforms originate from the fraction of the emitted optical pulseenergy that is reflected or backscattered by an object 16 located atrange R from the LiDAR system 10, and which is in the field of view(FoV) of the receiver optics 18. In this non-limiting example, a controland processing unit 20 controls the operation of both optical emitter 12and optical receiver 14 modules. Among other things, the control processmay synchronize the emission of each individual optical pulse with thestart of the sampling and ND data conversion of the return signalcollected by the receiver module 14. A digital clock 22 may be used togenerate clock signals for the control and processing unit 20 to ensureprecise timing of both modules, for example.

Optical Emitter Module

Upon reception of a trigger signal from the control and processing unit20, the driver electronics 24 may generate an electrical current pulsewhose duration lies in the ns range. The current pulse is then routed tothe light source 26 for emission of an optical pulse. The light source26 is generally a laser, but other types of optical sources, such aslight-emitting diodes (LEDs), can be envisioned without departing fromthe scope of the present disclosure. The use of semiconductor laserdiode assemblies now prevails in LiDAR systems. The laser diode assemblymay comprise a single-emitter laser diode, a multiple-emitter laserdiode, or even a two-dimensional stacked array of multiple-emitter laserdiodes. The specific type of light source integrated in a LiDAR system10 depends, inter alia, on factors such as the peak optical output powerrequired for successful ranging at the desired maximum range, theemission wavelength, and the device cost. Light sources such as fiberlasers, microchip lasers and even solid-state lasers find their way inLiDAR applications, particularly when no laser diode source exists atthe desired emission wavelength. The optical pulses pass through theemitter optics 28 before leaving the optical emitter module 12. Theemitter optics 28 shapes the optical pulses in the form of a beam havingthe desired propagation characteristics. The primary optical beamcharacteristics may be the beam divergence, the transverse size of thebeam irradiance profile at the exit aperture of the emitter module 12(e.g., for eye safety concerns), and the spatial beam quality. Theemitter 28 and receiver optics 18 are generally boresighted so as theoptical beam path and the field of view of the receiver module 14overlap over a predetermined range interval.

Optical Receiver Module

The return optical signals collected by the receiver optics 18 may passthrough a narrowband optical filter 30 for removal of the parasiticbackground light before impinging on the sensitive surface of aphotodetector 32. The photodetector 32 is generally an avalanche or PINphotodiode, or a 1D or 2D array of such photodiodes, with materialcomposition suited to the wavelength of the optical pulses. The currentfrom the photodetector 32 may then fed to a transimpedance (current tovoltage) amplifier 34. Also, the signal may or may not be pre-amplifiedas an APD typically has an internal current multiplication gain whichmay be sufficient.

The amplifier circuit may comprise a matched filter to limit theelectrical bandwidth of the optical receiver module 14. The control andprocessing unit 20 may control the amplifier gain to ensure that thesignal amplitude fits within the input voltage dynamic range of the A/Dconverter 36. It is known in the art that other amplifier configurationscould be used as well, such as a logarithmic amplifier or a set ofamplifiers mounted in parallel, each amplifier having a fixed gain. TheA/D converter 36 digitizes the input voltage signals at a sampling rateof typically several tens of MS/s (mega-samples per second) to a fewthousands of MS/s. The time period between two consecutive digitalsampling operations defines the extent of the so-called range bins ofthe system 10, when expressed in units of distance.

In many cases the output of the LiDAR system may be used by autonomouscomputer-based processes, e.g., to make navigation or mobility decisionsin autonomous vehicle applications. In some cases, a user may operatethe system 10 and receive data from it through the user interfacehardware 38. For instance, the measured range to the targeted object 16and/or a more detailed 3D map of the field of view may be displayed indigital form on a liquid-crystal or plasma visual display 40. Inaugmented reality applications, the detailed 3D map data may be combinedwith high-definition image data, e.g., from a high-definition digitalcamera (not shown), in order to allow virtual objects/elements to beplaced in a virtual environment displayed on the display 40.

Vehicles of all types now use LiDAR to determine which obstacles arenearby and how far away they are. The 3D maps provided by LiDARcomponents not only detect and position objects but also identify whatthey are. Insights uncovered by LiDAR also help a vehicle's computersystem to predict how objects will behave, and adjust the vehicle'sdriving accordingly.

Semi- and fully-autonomous vehicles may use a combination of sensortechnologies. This sensor suite could include Radar, which providesconstant distance and velocity measurements as well as superiorall-weather performance, but lacks in resolution, and struggles with themapping of finer details at longer ranges. Camera vision, also commonlyused in automotive and mobility applications, provides high-resolutioninformation in 2D. However, there is a strong dependency on powerfulArtificial Intelligence and corresponding software to translate captureddata into 3D interpretations. Environmental and lighting conditions maysignificantly impact camera vision technology.

LiDAR, in contrast, offers precise 3D measurement data over short tolong ranges, even in challenging weather and lighting conditions. Thistechnology can be combined with other sensory data to provide a morereliable representation of both static and moving objects in thevehicle's environment.

Hence, LiDAR technology has become a highly accessible solution toenable obstacle detection, avoidance, and safe navigation throughvarious environments in a variety of vehicles. Today, LiDARs are used inmany critical automotive and mobility applications, including advanceddriver assistance systems and autonomous driving.

In many autonomous driving implementations, the main navigation systeminterfaces with one or a few LiDAR sensors. It is desirable that theLiDAR sensor(s) offer high ranges and high resolutions in order tosupport functions such as localization, mapping and collision avoidance.In terms of localization, the first step of environment perception forautonomous vehicles is often to estimate the trajectories of thevehicle. Since Global Navigation Satellite System (GNSS) are generallyinaccurate and not available in all situations, the SimultaneousLocalization and Mapping (SLAM) technique is used to solve that problem.In terms of collision avoidance, a long detection range at cruisingspeed potentially provides sufficient time to react softly in case of anobstacle detection. For example, for standing users inside a shuttle, asafe and comfortable deceleration of 1.5 m/s² may be desirable. As anexample, at 40 km/h, and at 1.5 m/s² deceleration, a distance of 47 m isneeded to stop the shuttle, assuming a 0.5 s reaction time.

Many autonomous shuttles today rely on a long-range mechanical-scanningLiDAR sensor that is placed on top of the shuttle.

FIG. 3 shows an example of an autonomous driving vehicle 50 configuredas a multi-person shuttle with a conventional mechanical-scanning LiDARsensor 52 mounted on the top of the shuttle near the front of theshuttle. However, as discussed earlier, mechanical-scanning devices,such as a gimbal or spinning disks or polygons often come withresolution issues, maintenance issues, assembly issues and/ortemperature dependence issues.

Therefore, it would be desirable to provide LiDAR systems with solidstate scanning devices that avoid or at least mitigate one or more ofthese issues.

In terms of range and resolution, it is generally desirable to providedetectability at greater range and sufficient resolution to be able toaccurately categorize detected objects.

FIG. 4 shows examples of external objects having features above groundlevel that can pose detection problems for conventional LiDAR systems inautonomous driving or mobility applications. In particular, the examplesinclude a flatbed vehicle, a tail loading lift, a parking gate and ahanging trashcan. Such objects are potentially problematic because theyare not laying on the ground and have a relatively narrow verticalprofile at some point above the ground that must be avoided in order toavoid a collision.

As another aspect of collision avoidance, a LiDAR system with aside-looking field of view (FoV) can potentially be useful for turningassistance, particularly on larger vehicles, such as trucks or buses.For example, FIG. 5 shows an example of a potential use for a wide-angleside-looking LiDAR system for turning assistance on a freight truck. Inthis example, the LiDAR system 62 has a FoV that extends oversubstantially 180° and a range that extends in front of and behind theright side of the truck 60, which can help avoid a collision with acyclist 64 or moving objects or people within the FoV during a righthand turn. For example, if a moving object is detected within the FoV inadvance of, or while executing a right hand turn, the driver of thetruck may be alerted to the detection and/or the navigation system, ifpresent, may autonomously take mitigating steps, e.g., by activating thebrakes or steering away.

FIG. 6 shows another example of a potential use for a wide-angle LiDARsystem for blind spot coverage on a bus 70. In particular, in thisexample, the bus 70 has a wide-angle LiDAR system that includes a firstLiDAR sensor 72 providing a wide-angle FoV in front of the bus 70 andtwo side mounted LiDAR sensors 74 and 76 mounted on the left and rightsides of the bus, respectively, which have FoVs that cover the commonblindspots at the front and sides of a bus.

FIGS. 7A, 7B and 7C show top, side and front views, respectively, of ashuttle vehicle 80 having a wide-angle LiDAR system that include a highside-mounted wide-angle LiDAR sensor 82,84 on each side of shuttle 80.Each LiDAR sensor 82, 84 has a FoV 86,88 that extends in a verticaldirection substantially 90° from the horizon to the ground and in ahorizontal direction substantially 180° from the rear of the shuttle tothe front of the shuttle. It is noted that in FIG. 7C only the left sidemounted LiDAR sensor 84 is shown in order to avoid clutter in thedrawing. In the following description, the following nomenclature hasbeen adopted: the horizontal and vertical directions will be designatedwith the Cartesian coordinates x and y, respectively, and the directionperpendicular to the x-y plane is designated with the Cartesiancoordinate z.

Referring to FIG. 7C, since the position on the vehicle is at the top,the vertical FoV of the LiDAR sensor 84 points primarily towards theground 90.

FIG. 8 shows an example of a uniform vertical angular distribution of150 points extending over substantially 90° from the horizon to theground 90 thereby providing a uniform vertical resolution of 0.6°. Ifthe LiDAR system were designed with a uniform horizontal resolutions of0.2° and a uniform vertical resolution of 0.6°, the FoV, which coverssubstantially 180° horizontal and 90° vertical would require a total of135,000 pixels, most of which would be pointing at the ground near thevehicle.

For example, FIGS. 9A and 9B show example plots of simulated LiDAR pixeldata for the LiDAR sensor 84 having the uniform vertical angulardistribution of FIG. 8 with a 0.5 m×1.8 m target at distances of 15 mand 5 m, respectively. A pedestrian, such as the pedestrian 92 shown inFIG. 7C may be represented by a 0.5 m×1.8 m target. As shown in FIGS. 9Aand 9B, at distances of 15 m and 5 m respectively, the LiDAR sensor 84would have 121 and 899 pixels, respectively, on the pedestrian 92. Thislevel of coverage is typically not necessary for objectdetection/classification, which can typically be done reliably with asfew as 3-15 pixels.

In order to cover the same vertical FoV, i.e., substantially 90° fromthe horizon to the ground 90, while having relatively higher verticalresolution in certain parts of the vertical FoV and relatively lowervertical resolutions in other parts of the vertical FoV, the inventorsof the present disclosure have conceived of utilizing a non-uniformvertical angular distribution of scanning beams, thereby providingnon-uniform vertical resolution.

For example, FIG. 10 shows an example of a non-uniform vertical angulardistribution of 32 points extending over substantially 90° from thehorizon to the ground 90 defined by the following source code:

LL = 1:31;  RES_NL = 2.90802E−01 * exp(1.10636E−01*LL); V_NL(1) = 0;  for iV = 1:length(LL)    V_NL(iV+1) = V_NL(iV) + max([0.5RES_NL(iV)*1.08129]);   endfor

It should be noted that is merely one example of a non-linear functionthat may be used to generate a non-uniform angular distribution.Moreover, a person of ordinary skill in the art will recognize that thechoice of the distribution and the number of points over a given angularrange may vary depending on performance requirements, such as theminimum required vertical resolution, the minimum number of points on atarget of a given size at a given distance, etc.

FIG. 11 shows an example of a segmented FoV having areas of non-uniformvertical resolution based on the non-uniform vertical angulardistribution of FIG. 10 over substantially 90° from the horizon to theground in the vertical directly y and uniform horizontal resolution oversubstantially 180° in the horizontal direction x.

FIGS. 12A, 12B and 12C show example plots of simulated LiDAR pixel datafora LiDAR system having the non-uniform vertical angular distributionof FIGS. 10 and 11 with a 0.5 m×1.8 m target at distances of 5 m, 15 mand 50 m, respectively. As shown in FIGS. 12A, 12B and 12C, at distancesof 5 m, 15 m and 50 m, respectively, the LiDAR sensor 84 having thenon-uniform vertical angular distribution of FIGS. 10 and 11 would have90, 18 and 4 pixels, respectively, on a 0.5 m×1.8 m target (e.g., thepedestrian 92 of FIG. 7C). This level of coverage is generallysufficient for object detection/classification, which, as noted earlier,can typically be done reliably with as few as 3-15 pixels on a target.

A segmented FoV with uniform horizontal resolution and non-uniformvertical resolution can potentially be realized in many ways. Forexample, non-uniform magnification optics may be used either alone or incombination with a beam steering device in order to achieve a FoV withsuch properties.

FIG. 13 shows an example of a LiDAR system 1300 having non-uniformmagnification optics 1302 that amplifies the angular distribution of anoptical emitter module 1306 in accordance with an embodiment of thepresent disclosure. A number of components of the LiDAR system 1300 havebeen omitted from the illustration in the interest of clarity. In thisspecific example, the magnification optics 1302 magnifies the maximumangular range of approximately ±30° achievable by the emitter module1306 to approximately ±180°, thereby increasing the effective FoV of theLiDAR system 1300. In FIG. 13 , the LiDAR system 1300 includes a cover1304 that protects the magnification optics 1302. In someimplementations such a cover may be unnecessary and therefore may beomitted. In some embodiments, the magnification optics 1302 may have animage point distribution function that is non-linear relative to avertical field angle of object points in the FoV. For example, in someembodiments the magnification optics 1302 may provide a non-uniformvertical resolution over the vertical field of view similar to thesegmented FoV 1100 shown in FIG. 11 .

In some embodiments, non-uniform magnification optics, such as thenon-uniform magnification optics 1302 shown in FIG. 13 may be used toprovide a non-uniform distribution of uniformly distributed steeringangles from a beam steering component (not shown) that may be part ofthe emitter module 1306 or as a separate component located between theemitter module 1306 and the magnification optics 1302.

For example, returning to the segmented FoV 1100 shown in FIG. 11 , the32 non-uniformly distributed vertical angles can potentially be groupedinto four groups of eight angles each, wherein each successive group hasa lower average vertical resolution, as indicated at 1101, 1102, 1103and 1104 in FIG. 11 . This type of distribution of 32 vertical anglescan be realized using an 8 channel laser light source, a beam steeringdevice capable of steering each of the 8 laser channels in 4 verticaldirections, and non-uniform magnification optics to non-uniformlydistribute the 8 laser channels in each of the 4 vertical directions.For example, such a beam steering device could potentially beimplemented with a non-mechanical beam steering device such as a liquidcrystal polarization grating (LCPG). By “non-mechanical” is meant a beamsteering device that performs beam steering without mechanicaldisplacement or movement of the component performing the beam steeringoperation.

LCPGs, with nearly ideal diffraction efficiencies (>99.5%) have beenexperimentally demonstrated over a wide range of grating periods,wavelengths (visible to near-IR), and areas. Each polarization gratingstage can double the maximum steered angle in one dimension withoutmajor efficiency reductions, so very large steered angles are possible(at least to ±40° field of regard). The structure at the heart of thesedevices is a polarization grating (PG), implemented using nematic liquidcrystals. The nematic director is a continuous, in-plane, bend-splaypattern established using a UV polarization hologram exposingphoto-alignment materials. When voltage is applied, the director orientsout of plane, effectively erasing the grating. A single LCPG stage canbe considered the key component with three possible directions (±6 and0°), but additional steering angles are possible by stacking LCPGstages.

In another example of implementation, the beam steering device includesone or more LCPG stages, where each stage includes an LC switch and apassive grating. This configuration allows two possible steeringdirections.

It should be noted that an LCPG is merely one example of anon-mechanical beam steering device that may be used in some embodimentsof the present disclosure. Other non-limiting examples of beam steeringdevices, such an optical phased arrays (OPAs) or microelectromechanicalsystems (MEMS) that may be utilized in some embodiments of the presentdisclosure are described, for example, in Paul F. McManamon, AbtinAtaei,

“Progress and opportunities in optical beam steering,” Proc. SPIE 10926,Quantum Sensing and Nano Electronics and Photonics XVI, 1092610 (29 May2019), which is incorporated herein by reference in its entirety.

FIG. 14 shows an example of a digital beam steering componentimplemented using a LCPG 1400 in accordance with an embodiment of thepresent disclosure. In this example, the LCPG 1400 includes sevenstacked optical plates forming three steering stages and is configuredfor use with a laser that has a linear polarization. The optical platesare either a Liquid Crystal (LC) or a polarization grating (PG). Theassembly is configured to achieve both vertical steering angles andhorizontal steering angles. The LCPG has a nominal FoV of 7.5°×6.0°(Horizontal×Vertical) for a given steering direction or “tile” and afull FoV of 120°×24°.

FIG. 15 shows an example of two-dimensional (2D) beam steering anglesthat are possible using the LCPG beam steering element 1400 of FIG. 14 .It is noted that the LCPG beam steering element 1400 is capable ofsteering in 14 different horizontal directions and 4 verticaldirections. Each pair of a horizontal direction and a vertical directionin which the LCPG is capable of steering may be referred to as a “tile”in the full FoV of the LCPG. In this case, the LCPG 1400 is capable ofsteering in any one of 14×4 tiles, each with a nominal optical field ofview of 7.5°×6.0°, thereby providing a full FoV of 120°×24°. In thetable shown in FIG. 15 , minimum and maximum angles of each tile'sHorizontal FoV is shown above the addresses of the tiles, and theminimum and maximum angles of each tile's Vertical FoV is shown at theleft of the tiles addresses. For example, tile 6 has minimum and maximumangles of Horizontal FoV equal to −14.6° and −7°, respectively, andminimum and maximum angles of Vertical FoV equal to 6° and 12°,respectively.

However, the emission and reception efficiencies of the LCPG 1400 arenot constant with steering angle. FIG. 16 shows an example plot ofemission and reception efficiencies vs. horizontal steering angle forthe LCPG beam steering element 1400 of FIG. 14 . Emitter efficiency isthe top curve (small dots) and receiver efficiency is the bottom curve(large dots). The difference between the emission and receptionefficiencies is due to polarizer transmission, which may be 90%efficient.

Since emission and reception efficiencies drop off at higher horizontalsteering angles, in the following example only the center 8×4 tiles ofthe LCPG 1400 are utilized for horizontal and vertical steering. Inother implementations, more or fewer horizontal tiles may be used forhorizontal steering to provide a wider or narrower horizontal steeringrange. It is also noted that, since not all tiles of the LCPG 1400 areutilized in the current embodiment, in other embodiments an LCPG withfewer horizontal steering stacks may be utilized, which couldpotentially reduce cost and provide a gain in efficiency, and thereforein range.

FIG. 17 shows tables of examples of non-uniform steering angleconfigurations and corresponding geometric distances at heights of 2.5 mand 3.5 m for the LiDAR system 84 of FIG. 7C configured to provide thesegmented non-uniform FoV 1100 of FIG. 11 . In this case, magnificationoptics having non-uniform magnification in the vertical direction willbe utilized to non-uniformly distribute the substantially uniformvertical steering angles provided by the four vertical steeringdirections (tiles) of the LCPG 1400. Tables 2 and 3 in FIG. 17 give thedistance to the ground 90 or the range if the pixel does not reach theground 90. It is noted that, of the 32 vertical angles, 31 pixels arepointed at the ground 90, in other words these angles will intersect theground plane at some distance from the source. Only the zero degreeangle relative to the horizon will not intersect the ground plane. Thefour columns of each table correspond to each of the four verticalsteering directions of the LCPG tiles. The eight rows of each tablecorrespond to the eight channels of the eight-channel laser light sourcethat is used in this example. Nominal magnification for the four LCPGtiles is approximately 0.66×, 1.33×, 3.33× and 10×, respectively,calculated based on the increase in vertical magnification relative tothe nominal uniform vertical angular resolution of 0.75° for 32 anglesuniformly distributed over the 24° FoV of the LCPG.

FIG. 18 shows a top down view of an example of a LiDAR system 1800 withnon-uniform magnification optics 1802 that may be used to implement thesegmented FoV 1100 with non-uniform vertical resolution and uniformhorizontal resolution of FIG. 11 . A number of components of the LiDARsystem 1800 have been omitted from the illustration in the interest ofclarity. As shown in FIG. 18 , the LCPG beam steering device 1806 iscapable of steering optical beams from the 8-channel laser light source1810 in any of eight horizontal directions that are generally uniformlydistributed between −29.7° and 29.7°. The magnification optics 1802 isconfigured to provide substantially uniform magnification in thehorizontal direction, and distributes the optical beams over eightsegments that are generally uniformly distributed between −90° and 90°.It is noted that these eight segments correspond to the eight “columns”in the segmented FoV 1100 of FIG. 11 . Although the correspondingreception path for light reflected off object(s) in the FoV back to theLiDAR system 1800 is not shown in FIG. 18 in order to avoid clutteringthe drawing, the magnification optics 1802 and LCPG beam steering device1806 essentially act in reverse for optical beams of reflected lightthat is received from the FoV of the LiDAR system 1800. In particular,the magnification optics 1802 takes in reflected light from the outermagnified FoV and de-magnifies it via refraction onto a reducedhorizontal angular range. The received reflected light beams of areduced horizontal angular range are then directed onto the LCPG beamsteering device 1806, which, via the reception optics 1812, directs themonto sensor unit 1814

The LiDAR system 1800 has the wide-angle magnification optics 1802, aprotective cover 1804 that may not be present in some embodiments, abeam steering device 1806, which in this embodiment is implemented bythe 8×4 tiles of the LCPG 1400 of FIG. 14 , emission optics 1808, an8-channel laser light source 1810, reception optics 1812 and a sensorunit 1814.

FIG. 19 shows a side on view of the LiDAR system 1800 of FIG. 18 showingthe non-uniform vertical steering angles resulting from the non-uniformmagnification optics 1802. In this case, because the vertical emissionangles extend substantially 90° from the horizon to the ground 90, themagnification optics 1802 could potentially be implemented with only thebottom-half of a hemispherical objective lens. In FIG. 19 , thenon-uniform vertical distribution of the vertical steering anglesimparted by the LCPG beam steering element 1806 in the verticaldirection is shown for two of the eight channels of the eight-channellaser light source 1810. In particular, FIG. 19 shows the non-uniformvertical distribution of the four vertical steering angles imparted bythe four tiles of LCPG beam steering element 1806 for laser channels 1and 8, which cover the full range of vertical steering angles oversubstantially 90° between the horizon and the ground 90 in theembodiment shown in FIG. 7C. More specifically, as shown in FIG. 19 ,the generally uniformly distributed vertical steering angles imparted tolaser light from laser channel 1 by the LCPG beam steering device 1806,which nominally cover four vertical angles over a range of approximately24°, are non-uniformly distributed at angles of 0°, 4.6°, 15° and 40°relative to the horizontal. In this way, the 24° generally uniformlydistribution of the distributed vertical steering angles imparted by theLCPG beam steering device 1806 to the laser light from laser channel 1is non-uniformly magnified to 40° by the non-uniform magnificationoptics 1802. Similarly, the generally uniformly distributed verticalsteering angles imparted to laser light from laser channel 8 by the LCPGbeam steering device 1806, which nominally cover four vertical anglesover a range of approximately 24°, are non-uniformly distributed atangles of 3.8°, 13.1°, 35.6° and 90° relative to the horizontal. In thisway, the 24° generally uniformly distribution of the distributedvertical steering angles imparted by the LCPG beam steering device 1806to the laser light from laser channel 8 is non-uniformly magnified to86.2° by the non-uniform magnification optics 1802. Here it is notedthat the non-uniform distribution of vertical steering angles for laserchannels 1 and 8 correspond to the configuration angles in the first andlast rows, respectively, of Table 1 in FIG. 17 .

In the LiDAR system 1800 shown in FIGS. 18 and 19 , two of the laserchannels may be activated or “fired” at the same time, such that opticalimpulses from alternating pairs of laser channels are transmittedaccording to a particular firing sequence. In such embodiments, thesensor unit 1814 may be implemented by an array of avalanche photodiodes(APDs) arranged in a 32×2 array configuration, whereby each of the twosets of 32 APDs measures one horizontal segment of the segmented FoV1100 for one of the two lasers fired at a given time.

FIG. 20 shows an example of such a firing sequence for the 8-channellaser light source 1810 and the corresponding configuration of thesensor unit 1814 of the LiDAR system 1800 of FIGS. 18 and 19 . Inparticular, in this example, firing sequence is such that the followingpairs of laser channels are fired together: 1 and 5, 2 and 6, 3 and 7, 4and 8. Other variations are possible and are contemplated within thescope of the present disclosure. In this configuration, laser channels1-4 optically correspond to the top 32×1 APDs of the sensor unit 1814and laser channels 5-8 optically correspond to the bottom 32×1 APDs ofthe sensor unit 1814. Here it is noted that by utilizing 32 APDs tomeasure each of eight generally uniform horizontal segments across ahorizontal FoV that spans substantially 180°, the resulting generallyuniform horizontal resolution is approximately 0.7°.

FIG. 21 shows an example of an accumulation strategy for the segments ofthe segmented FoV 1100 represented by the steering angle configurationsof the LCPG beam steering device 1806 of the LiDAR system 1800 of FIGS.18 and 19 . In this example, the analysis starts with a singleaccumulation in each tile. Additional acquisitions may then be done inactive tiles with potentially more accumulations being done in tileswithin specific regions of interest. In this example, more accumulationsare performed in the top row of tiles, which represents the segmentshaving the highest vertical resolution (i.e., the top eight segments ofthe segmented FoV 1100 of FIG. 11 ), with progressively feweraccumulations in each subsequent row of tiles.

In the example shown in FIG. 21 , there are an equal number ofaccumulations in the horizontal direction of each row of tiles. However,in some cases, depending on the magnification factor and the effectiveaperture, the accumulations across the horizontal axis could be changed,e.g., to favor the front direction of the vehicle by performing moreaccumulations in the horizontal direction towards the front of thevehicle. For example, FIG. 22 shows an example of another accumulationstrategy for the segments of the segmented FoV with unequalaccumulations along the horizontal direction.

In the example LiDAR system 1800 shown in FIGS. 18 and 19 , emission andreception functions utilize the same LCPG beam steering device 1806 andmagnification optics 1802. However, other configurations of the opticalemission and reception paths are possible and are contemplated withinthe scope of the present disclosure.

For example, FIG. 23 shows a top down view of an example of a LiDARsystem 2300 with non-uniform magnification optics in which the emittermodule and the reception module have separate magnification optics. Inparticular, in the example embodiment shown in FIG. 23 , a laser lightsource 2310, emission optics 2308 and an emission beam steering device2306 are behind a first non-uniform magnification optics 2302 andprotective cover 2304, and a sensor unit 2314, reception optics 2312 andreception beam steering device 2307 are behind a second non-uniformmagnification optics 2303 and protective cover 2305. This configurationmay provide more range, but potentially at the cost of a larger physicalfootprint. In some embodiments, a single protective cover may be used tocover both magnification optics 2302 and magnification optics 2303. Inother embodiments, one or both of the protective covers 2304 and/or 2305may be omitted. As in the previous examples, a number of components ofthe LiDAR system 2300 have been omitted from the illustration in theinterest of clarity.

As another example, FIG. 24 shows a top down view of an example of aLiDAR system with non-uniform magnification optics in which the opticalemitter and reception paths are co-axial. In particular, in the exampleembodiment shown in FIG. 24 , an optical path that includes a protectivecover 2404, outer non-uniform magnification optics 2402, a bead steeringdevice 2406 and inner uniform magnification optics 2416 is then split bya polarization beam splitter (PBS) 2418 towards an emitter module thatincludes emission optics 2420 and a laser light source 2410 and areception module that includes reception optics 2422 and a sensor unit2414. This configuration may provide a smaller footprint than theprevious embodiments and may provide better short range performance, butlosses due to the PDS 2418 may limit its range. In some embodiments, theprotective cover 2404 may be omitted. As in the previous examples, anumber of components of the LiDAR system 2400 have been omitted from theillustration in the interest of clarity.

In the examples discussed above with reference to the LiDAR system 1800of FIGS. 18 and 19 , the beam steering device 1806 was implemented withan LCPG device. However, as discussed previously, in other embodimentsbeam steering may be done via a different type of beam steering deviceand/or in conjunction with two or more different types of beam steeringdevice. In still other embodiments, the light signals magnified vianon-uniform magnification optics may not receive active optical steeringprior to being non-uniformly magnified via the non-uniform magnificationoptics.

For example, FIG. 25 shows a top down view of an example of a LiDARsystem 2500 with non-uniform magnification optics 2502 and a beamsteering device implemented by a MEMS device 2516. In particular, theLiDAR system 2500 includes a laser light source 2510 arranged totransmit laser light onto MEMS device 2516, which reflect the laserlight through emission optics 2508 and from there the emitted light beamis magnified by non-uniform magnification optics 2502. As in theprevious example, a protective cover 2504 overs magnification optics2502, but the protective cover 2504 may be omitted in some embodiments.On the reception side, a sensor unit 2514 receives reflected opticalsignals from the LiDAR system's FoV through magnifying optics 2502 andreception optics 2512. The MEMS beam steering device 2516 may be anytype of MEMS device that is capable of steering an optical signal withsufficient energy density to be useful for LiDAR applications. Forexample, MEMS micro-mirrors can steer light continuously by at least±30°. As in the previous examples, a number of components of the LiDARsystem 2500 have been omitted from the illustration in the interest ofclarity.

In some embodiments, two beam steering devices, such as an LCPG beamsteering device and a MEMS beam steering device, may be used inconjunction with one another to provide coarse and fine scanningfunctions. For example, a MEMS beam steering device may be used for finescanning with a coarse scanning segment of an LCPG beam scanning device.

As one example, FIG. 26 shows a top down view of the LiDAR system 2500of FIG. 25 with an added LCPG beam steering device 2506 between theemission optics 2508 and the non-uniform magnification optics 2502. Itis noted that in this example the LCPG beam steering device 2506 alsoprovides reception beam steering functionality for sensor unit 2514. Inthis example, the LCPG beam steering device 2506 may provide coarsehorizontal beam steering between and the MEMS beam steering device 2516may provide fine scanning within each coarse segment of the LCPG beamsteering device 2506. In some embodiments, the MEMS beam steering device2516 may be implemented by a 1-D resonant MEMS device.

FIG. 27 shows a top down view of an example of a LiDAR system 2700 withnon-uniform magnification optics 2702 and a FLASH structure for opticalbeam dispersal. In particular, in the LiDAR system 2700 shown in FIG. 27, the emitter module includes a laser light source 2710 and emissionoptics 2706 that horizontally diffuse laser light from the laser lightsource 2710 over first horizontal angular range 2720 that is thenfurther magnified to a second wider horizontal angular range 2722 by themagnification optics 2702. The laser light source 2710 may be amulti-channel laser light source similar to the 8 channel laser lightsource 1810 of the LiDAR system 1800 of FIGS. 18 and 19 . In suchembodiments, the non-uniform magnification optics 2702 may non-uniformlydistribute the horizontally diffused laser light from each of thedifferent laser channels at non-uniformly spaced vertical angles toprovide a non-uniform vertical resolution over the FoV of the LiDARsystem 2700. On the reception side, a sensor unit 2714 receivesreflected optical signals from the LiDAR system's FoV through magnifyingoptics 2702 and reception optics 2712. Such embodiments may be wellsuited to lower power/lower range applications, such as in mobiledevices where high power LEDs may even be used rather than a higherpowered laser light source like the laser light source 2710. As in theprevious example, a protective cover 2704 overs magnification optics2702, but the protective cover 2704 may be omitted in some embodiments.As in the previous examples, a number of components of the LiDAR system2700 have been omitted from the illustration in the interest of clarity.

FIG. 28 shows another example of a LiDAR system 2800 with non-uniformmagnification optics 2802 according to an embodiment of the presentdisclosure. In addition to the non-uniform magnification optics 2803,the LiDAR system 2800 of FIG. 28 includes an emission unit 2820, asensor unit 2814 and a computer device 2830. The emission until 2820 isconfigured for emitting an optical signal that illuminates at least partof a FoV of the LiDAR system 2800. In particular, the emission unit 2820emits an optical signal that illuminates a field of emission (FoE) thatat last partially overlaps with a field of reception (FoR) from whichthe sensor unit receives optical signals. The FoV is defined as the areaof overlap between the FoE and FoR. In operation, an optical signal 2840emitted by emission unit 2820 is refracted by non-uniform magnificationoptics such that there is a non-linear relationship between the angularorientation of the emitted optical signal 2840 along at least one axis(e.g., a vertical axis and/or horizontal axis) relative to the angularorientation of an outwardly emitted signal 2842 resulting from therefraction (angular magnification) of the non-uniform magnificationoptics 2802. For example, the magnification optics may have an imagepoint distribution function that is non-linear relative to a verticalfield angle of object points in the FoV. In this embodiment, themagnification optics 2802 is configured for receiving an optical signal2860 that is a version of the emitted optical signal 2842 reflected fromat least one object in the FoV (as indicated at 2850 in FIG. 28 ). Inthis embodiment, the magnification optics refract the received reflectedoptical signal 2860 towards the sensor unit 2814, which is configuredfor processing the received optical signal 2862 and outputting a depthmap of the FoV. For example, if the magnification optics 2802 has animage point distribution function that is non-linear relative to avertical field angle of object points in the FoV, then the depth map mayhave at least one substantially expanded zone and at least onesubstantially compressed zone in the vertical direction. In someembodiments, the sensor unit 2814 may receive optical signals and obtainoptical measurements based thereupon, but the processing and outputtingof the depth map may be carried out by computing device 2830 based uponthe measurements obtained by sensor unit 2814. In some embodiments, thecomputer device 2830 may also provide control signals to emission unit2806 in order to coordinate the emission and reception functions.

In some embodiments, the magnification optics 2802 comprises anobjective lens 2803, wherein the sensor unit 2814 comprises a pluralityof sensor elements placed in an image plane of the objective lens 2803.For example, the sensor unit 2814 may include an array of APDs asdescribed earlier with reference to FIG. 20 . In such embodiments, ifthe magnification optics 2802 has an image point distribution functionthat is non-linear relative to a vertical field angle of object pointsin the FoV, then a number of sensor elements per degree of verticalfield angle may differ over portions of the FoV by more than 10%relative to the average number of sensor elements per degree of verticalfield angle over the total FoV in the vertical direction.

In some embodiments, if the magnification optics 2802 has an image pointdistribution function that is non-linear relative to a vertical fieldangle of object points in the FoV and the depth map may have at leastone substantially expanded zone and at least one substantiallycompressed zone in the vertical direction, then the objective lens andthe plurality of sensor elements may be configured such that, in eachsubstantially expanded zone, a number of sensor elements per degree ofvertical field angle is greater than the average number of sensorelements per degree of vertical field angle over the total FoV in thevertical direction and, in each substantially compressed zone, thenumber of sensor elements per degree of vertical field angle is lessthan the average number of sensor elements per degree of vertical fieldangle over the total FoV in the vertical direction.

In the LiDAR system 2800 shown in FIG. 28 , the emitted optical signal2840 passes through the magnification optics 2840 and is refractedthereby before illuminating at least part of the FoV. However, in otherembodiments, magnification optics may only be used to receive opticalsignals from the FoV or magnification optics different from those usedfor optical signal reception may be used for emission.

In some embodiments, the LiDAR system 2800 may include innermagnification optics between the emission module 2820 and themagnification optics 2802, such that the optical signal 2842 passesthrough two magnification optics before illuminating at least part ofthe FoV.

In some embodiments, the depth map is an original depth map, wherein thesensor unit or the computing device 2830 is configured for correctingthe original depth map for the non-linear distribution function toproduce a new depth map in which the substantially compressed zone inthe original depth map is expanded in the new depth map and in which thesubstantially expanded zone in the original depth map is compressed inthe new depth map.

In some embodiments, the new depth map comprises pixels and wherein atleast some of the pixels in a portion of the new depth map correspondingto an expanded version of a substantially compressed zone in theoriginal depth map are interpolated pixels.

In some embodiments, the sensor unit is configured for processing thedepth map to determine a location of the object in the FoV and adistance to the object in the FoV.

In some embodiments, the LiDAR system 2800 further includes a beamsteering unit 2806 for orienting the optical signal towards the FoV in aselected one of a plurality of directions. For example, the beamsteering unit 2806 may be part of the emission unit 2820 as shown inFIG. 28 , or it may be a component that is shared with the receptionpath to provide received beam steering between the magnification optics2802 and the sensor unit 2814.

In some embodiments, each of the steering directions is associated witha respective sub-area of the FoV.

In some embodiments, the beam steering unit 2806 is a solid-state beamsteering unit. For example, the beam steering unit 2806 may comprise anLCPG.

In some embodiments, the beam steering unit comprises a multi-stagesystem. For example, one stage of the multi-stage system may comprise anLCPG.

In some embodiments, the magnification optics is configured formagnifying a range of angles illuminated by the emitted optical signal.

In some embodiments, the emission unit 2820 is configured forcontrollably emitting a selected one of a plurality of optical beams asthe emitted optical signal 2840.

In some embodiments, each of the plurality of optical beams is orientedin a predetermined direction.

In some embodiments, the FoV comprises a vertical component and ahorizontal component, wherein the FoV spans at least 60 degrees in thevertical direction between horizon and ground.

In some embodiments, the FoV spans at least 150 degrees in thehorizontal direction.

In some embodiments, the image point distribution function issubstantially linear relative to a horizontal field angle of objectpoints in the FoV. In other embodiments, the image point distributionfunction of the magnification optics 2820 is non-linear relative to ahorizontal field angle of object points in the FoV. For example, theimage point distribution function may be symmetric relative to ahorizontal field angle of object points in the FoV.

FIG. 29 shows a flowchart of a method according to another embodiment ofthe present disclosure.

At step 2900 of the method a first image of a scene is captured via afirst sensor.

At step 2902, a second image of the scene is captured via a secondsensor different from the first sensor. The first and second imagesoverlap to include at least one common FoV. In some embodiments, thefirst image comprises pixels that are distributed in accordance with anon-linear image point distribution function relative to a field angleof object points of the FOV. In some embodiments, one of the first andsecond images is a depth map.

At step 2904, the first image is corrected based on said non-lineardistribution function to produce a third image.

AT step 2906, the second and third images are combined with each otherto produce a composite image including information from the second imageand information from the third image.

In some embodiments, the image point distribution function is non-linearin the vertical direction between horizon and ground.

In some embodiments, the image point distribution function has a maximumdivergence of at least ±10% compared to a linear distribution function,in the vertical direction.

In some embodiments, the image point distribution function issubstantially linear in the horizontal direction.

In some embodiments, the third image has more pixels than the firstimage.

In some embodiments, some of the pixels of the third image corresponddirectly to pixels of the first image and wherein other ones of thepixels of the third image correspond to interpolated versions of some ofthe pixels of the first image.

In some embodiments, the method may further include interpolating saidsome of the pixels of the first image to produce said other ones of thepixels in the third image.

In some embodiments, the other one of the first and second images is a2D camera image.

In some embodiments, the first sensor comprises an array of photodiodesand wherein the second sensor comprises a digital camera.

In some embodiments, the second image comprises pixels that aredistributed in accordance with a substantially linear distributionfunction relative to a field angle of object points of the FOV.

In some embodiments, the FOV comprises a vertical FOV and a horizontalFOV.

In some embodiments, the vertical FOV spans at least 60 degrees and thehorizontal FOV spans at least 150 degrees.

In some embodiments, the image point distribution function beingnon-linear relative to a field angle of object points in at least thevertical FOV.

In some embodiments, the image point distribution function is non-linearrelative to a field angle of object points in both the horizontal FOVand the vertical FOV.

In some embodiments, the composite image is an RGBD image.

In some embodiments, the first image comprises at least onesubstantially compressed zone and at least one substantially expandedzone, and wherein correcting the first image comprises at least one of(i) compressing the substantially expanded zone and (ii) expanding thesubstantially compressed zone, to produce the third image.

In some embodiments, capturing the second image of the scene at step2902 is carried out by sequentially capturing different subportions ofthe FOV as illuminated by an optical signal emitted in a controllabledirection.

FIG. 30 shows another example of a LiDAR system with non-uniformmagnification optics according to an embodiment of the presentdisclosure. In particular, FIG. 20 is a vertical cross-sectional view ofa LiDAR system using wide-angle magnification optics. A number ofcomponents of the LiDAR system 3000 have been omitted from theillustration in the interest of clarity.

The LiDAR system 3000 has a wide-angle magnification optics 3002 and alight sensor 3004. What is being shown in FIG. 30 is effectively thereceiver side of the LiDAR system 3000, it being understood that for afunctional LiDAR system to generate a three-dimensional LiDARrepresentation of the geographical area or scene, an emission side isrequired to generate a light signal that will illuminate thegeographical area and produce optical returns that are sensed by thereceiver side.

The wide-angle magnification optics achieves a wide-angle field of view.By “wide-angle’ is meant an optical aperture of at least 150 degrees insome axis, for example a horizontal axis. Preferably the angularaperture is close to 180 degrees. This is advantageous in automotiveapplications where the LiDAR system enables autonomous driving ordriving facilitation functions and 180 degrees of angular aperture wouldallow a wide enough view of the road. Note that in a number ofapplications of the LiDAR system, the angular aperture may be constantin all directions, such as in the horizontal direction or the verticaldirection. In other applications, the angular aperture may vary, forinstance it may be larger in the horizontal direction and narrower inthe vertical direction. The later variant is useful in automotiveapplications where a wide horizontal view of the road is important, buta wide vertical view of the road is not as essential.

The light returns that reach the lens 3002 are projected on the lightsensor 3004. The configuration of the lens 3002 is selected to adapt thelight projection on the light sensor 3004 in order to provideadvantages. Particularly, the lens 3002 is configured to project arepresentation of the scene conveyed by the light return by compressinga portion of that representation while expanding other portions. Forexample, a portion of the representation that may be expanded is onewhich is more susceptible to contain objects of interest, while aportion of the representation that may be compressed is one which isless susceptible to contain objects of interest. In automotiveapplications, where the LiDAR system 3000 has a view of the road, thecentral part of the field of view of the LiDAR system 3000 is whereobjects of interest are likely to reside, such as automobiles,pedestrians or obstacles. The peripheral part of the field of view isless likely to contain objects of interest. As a car drives on a road,most of the driving decisions are influenced by the what happens ahead,not on the side, hence it is important for the LiDAR system 3000 to havethe best visibility in that area.

However, there may be other applications where it is more important tohave a good peripheral vision than a central one. In such applications,the lens 3002 would be configured differently to manipulate the lightreturn such as to expand the peripheral area of the light return andcompress the central area of the light return.

The selective expansion and compression of the light return isaccomplished by selecting the lens geometry to achieve the desiredeffect. This is illustrated with greater detail at FIG. 30 . Assume forthe purpose of this example that the lens 3002 is a hemispherical lensand provides 180 degrees of optical aperture overall. The lens 3002receives optical returns over its entire outer surface and directs thosereturns towards the light sensor 3004. In this example ofimplementation, the light sensor is made up of an array of AvalanchePhotodiodes (APD) arranged in a suitable configuration. A data processor(not shown) receives the outputs of the various APDs, processes them toprovide a wide-angle three-dimensional representation of the scene infront of the lens. The representation of the scene would typically beexpressed as a series of points, where each point can be defined by X, Yand Z coordinates or by two angles, one in the horizontal plane, one inthe vertical plane and a distance dimension from a point of reference O.

The lens 3002 has a central area 3006 and a peripheral area 3008. Thecentral area 3006 receives a light return from an area S1 of the scene.The boundaries between S1 and S2 are conceptually shown as dotted lines3010 and 3012. In three dimensions the lines 3010 and 3012 form afrustum of a cone.

The central area of the lens 3006 provides a higher magnification thanthe peripheral area 3008. The practical effect of this arrangement is todirect the light of the return signal in the cone defined by lines 3010and 3012 over a larger surface area of the sensor 3004, than if themagnification would be the same across the lens 3002.

FIG. 31 is a schematical representation of the light sensor 3004,showing with concentric circles the surface area of the sensor overwhich the light return is spread, in the two scenarios, one where themagnification of the lens is constant and one where there is highermagnification at the center. D1 is the diameter of the circle associatedwith a constant magnification, while D2 is the circle associated withhigher magnification in the central area 3006. D2 is larger, whichimplies that the light information is spread over a larger surface ofthe light sensor 3004.

In LiDAR architectures using a flash optical illumination, where thelight return is received at once by the lens 3002, the approximateobject location in the scene is determined on the basis of the positionof the one or more light sensing elements on the light sensor 3004 thatrespond to the light return. When the light sensing elements are APDs,the position of the APDs that output a signal indicating the presence ofan object provides the approximate location of the object in the scene.Accordingly, by spreading the light information over a larger portion(the circle D2) of the light sensor 3004, a better resolution isobtained as more APDs are involved in the object sensing. Thus, it ispossible to tell with a higher level of precision the location in thescene where the detected objects reside.

Objectively, light received over the peripheral area 3008 is focused ona smaller portion of the light sensor 3004, which means that fewer APDsare available for sensing. This implies that the detection has lowerresolution, however, the peripheral area is less likely to containobjects of interest, hence the trade-off of increasing the resolution inthe center at the expense of reducing the resolution at the peripheryprovides practical advantages overall.

In a different LiDAR architecture, which uses a steerable illuminationbeam, the variable magnification lens 3002 also provides advantages. Inthe steerable beam architecture, the light emission can be steered toscan the scene and thus direct the light toward a particular area of thescene. A steerable beam architecture uses a beam steering engine whichcan be based on solid state components, mechanical components or acombination of both. Examples of solid-state components includeopto-electric plates that can change the angle of propagation of lightby applying a voltage. Example of mechanical components include MEMSmirrors that can change the orientation of a light beam.

FIG. 32 illustrates the architecture of a dual sensor system, includinga LiDAR system 3202 and an image system 3204. Sometimes, the LiDARsystem 3202 is referred to as “active” system, while the image system3204 is referred to as “passive” system. The LiDAR system outputs athree-dimensional representation of the scene while the image systemproduces a two-dimensional representation of the scene. It is known tomerge the outputs of the two systems in order to provide a 3D map of theenvironment. Typically, this is referred as a “sensor fusion” process.

The concept of sensor fusion between a LiDAR and an image is toattribute distance measurements to individual pixels or pixel groups inthe image. Hence, the 3D map can have a point cloud structure, whereindividual points are distributed in a space and each point has one ormore other attributes such as color, transparency, etc. Since a LiDARsystem operates typically at a lower resolution than an image system, itis also known to perform an upsampling operation when the LiDAR data ismerged with the image data, where distance information is derived andattributed to pixels or pixels groups for which the LiDAR system doesnot have a direct measurement. A technique which has been proposed inthe past is to rely of visual similarity in order to derive distancesimilarity. In other words, areas of the image which are visuallysimilar to an area for which a distance measurement has been obtainedfrom a LiDAR system, are assumed to be at the same or similar distancefrom a reference point. In this fashion, a three-dimensionalrepresentation from a lower resolution LiDAR system can be used with ahigh-density image to obtain a 3D map having a resolution higher thanthe resolution provided by the LiDAR system.

A practical approach in generating a 3D map is to determine which datapoints in the three-dimensional LiDAR representation, correspond towhich pixels or groups of pixels in the high-density image. In otherwords, a registration should be achieved such that a data point in theLiDAR representation and a corresponding pixel or group of pixelsrepresent the same object in the scene. Such registration operation ischallenging in instances where the three-dimensional LiDARrepresentation of the environment is non-uniform, for instance as aresult of using a variable magnification wide-angle lens, where someportions of the representation are at a higher resolution than others orotherwise distorted such that the distance from one data point toanother in the LiDAR representation is not necessarily the same as thedistance from one pixel to another in the image.

FIG. 33 is flowchart of a computer process which compensates thethree-dimensional representation for the distortion induced by thevariable magnification lens in order to create an undistortedrepresentation that is simpler to register with the image to create asound data point to data point correspondence. The process is performedby a computer device 3206 which receives at its inputs thethree-dimensional wide-angle representation of the scene from the LiDARsystem 3202 and the high-density image from the image system 3204,processes them and outputs a 3D map. The computer system has a CPU whichis programmed with software encoded on a non-transitory storage mediumto perform the data processing illustrated at FIG. 33 .

At step 3300 of the process the computer device compensates for thedistortion in the three-dimensional representation of the LiDAR data.Since the distortion model is known, namely the magnification pattern ofthe lens, the parts of the representation that have been distorted inrelation to other parts can be undistorted fully or in part. Examples ofdistortion correction include:

-   -   1. The portion of the image having a lower resolution can be up        sampled in order to equalize the resolution across the entire        representation. The up sampling can be done by interpolation        between data points in the lower resolution area. No        interpolation is performed in the area of the representation        that is at a higher resolution.    -   2. Expand the image in areas that have been compressed by using        the inverse of the magnification function of the lens. That will        expand areas of the image in order to produce a resolution        consistent with the lower resolution portion.

Alternatively, the image data can be distorted in a way which isconsistent with the distortion of the LiDAR three-dimensional data,allowing to register both data sets. One way to achieve the distortionis to use a magnification lens 3212 for the image sensor 3208 which hasthe same magnification pattern as the lens 3002. In this fashion bothdata sets can be registered to establish correspondence between the datapoints and eventual merge. Another option is to perform the distortionthrough data processing by the computer device 3206.

At step 3302, the compensated LiDAR data is merged with the image data.For example, the process described in the U.S. Pat. No. 10,445,928 inthe name of Vaya Vision, Ltd., the entire contents of which inincorporated herein by reference, can be used for that purpose.

In the dual sensor system architecture of FIG. 32 , the LiDAR system3206 and the image system 3204 have separate magnification optics 3002and 3212, respectively. FIG. 34 illustrates another example of anarchitecture for a dual sensor system that includes a LiDAR system 3402and an image system 3404 that share common magnifying optics 3402 forthe light sensor 3004 of the LiDAR system and the image sensor 3408 ofthe image system 3404. In this configuration, the three-dimensionalrepresentation of the LiDAR data and the image data captured by imagesensor 3408 may be subject to the same or similar distortion from themagnifying optics 3402, and therefore registration and an eventual mergebetween the two data sets may be accomplished more easily.

Furthermore, an architecture like that shown in FIG. 34 may have arelatively smaller physical footprint, which may be important inapplications where space is at a premium, such as in portable devices,e.g., smartphones or tablets. The merging of high resolution image dataand depth data obtained through an integrated LiDAR system may haveseveral uses in the mobile device context. For example, a higherresolution depth map resulting from merging depth data captured via thelight sensor 3004 with high definition image data captured image sensor3408 may be used for augmented reality applications, where the placementand interaction with virtual objects within an augmented reality spacemay rely on accurate and timely updated depth data, or in securityapplications where the addition of higher resolution depth data tofacial recognition applications may improve device security.

Certain additional elements that may be needed for operation of someembodiments have not been described or illustrated as they are assumedto be within the purview of those of ordinary skill in the art.Moreover, certain embodiments may be free of, may lack and/or mayfunction without any element that is not specifically disclosed herein.

Any feature of any embodiment discussed herein may be combined with anyfeature of any other embodiment discussed herein in some examples ofimplementation.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements, but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the inventive concept. Thisdescription should be read to include one or more and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.

Further, use of the term “plurality” is meant to convey “more than one”unless expressly stated to the contrary.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Circuitry, as used herein, may be analog and/or digital, components, orone or more suitably programmed microprocessors and associated hardwareand software, or hardwired logic. Also, “components” may perform one ormore functions. The term “component,” may include hardware, such as aprocessor, an application specific integrated circuit (ASIC), or a fieldprogrammable gate array (FPGA), or a combination of hardware andsoftware. Software includes one or more computer executable instructionsthat when executed by one or more component cause the component toperform a specified function. It should be understood that thealgorithms described herein are stored on one or more non-transitorymemory. Exemplary non-transitory memory includes random access memory,read only memory, flash memory or the like. Such non-transitory memorymay be electrically based or optically based.

As used herein, the term “substantially” means that the subsequentlydescribed parameter, event, or circumstance completely occurs or thatthe subsequently described parameter, event, or circumstance occurs to agreat extent or degree. For example, the term “substantially” means thatthe subsequently described parameter, event, or circumstance occurs atleast 90% of the time, or at least 91%, or at least 92%, or at least93%, or at least 94%, or at least 95%, or at least 96%, or at least 97%,or at least 98%, or at least 99%, of the time, or means that thedimension or measurement is within at least 90%, or at least 91%, or atleast 92%, or at least 93%, or at least 94%, or at least 95%, or atleast 96%, or at least 97%, or at least 98%, or at least 99%, of thereferenced dimension or measurement.

In case of any discrepancy, inconsistency, or other difference betweenterms used herein and terms used in any document incorporated byreference herein, meanings of the terms used herein are to prevail andbe used.

Although various embodiments and examples have been presented, this wasfor purposes of describing, but should not be limiting. Variousmodifications and enhancements will become apparent to those of ordinaryskill and are within a scope of this disclosure.

The invention claimed is:
 1. A method, comprising: capturing a firstimage of a scene via a first sensor; (ii) capturing a second image ofthe scene via a second sensor different from the first sensor; whereinthe first and second images overlap to include at least one commonfield-of-view (FoV); wherein the first image comprises pixels that aredistributed in accordance with a non-linear image point distributionfunction relative to a field angle of object points of the FoV; whereinone of the first and second images is a depth map; (iii) correcting thefirst image based on said non-linear distribution function to produce athird image; and (iv) combining the second and third images with eachother to produce a composite image including information from the secondimage and information from the third image.
 2. The method of claim 1,wherein the image point distribution function is non-linear in thevertical direction between the horizon and the ground.
 3. The method ofclaim 2, wherein the image point distribution function has a maximumdivergence of at least ±10% compared to a linear distribution function,in the vertical direction.
 4. The method of claim 2, wherein the imagepoint distribution function is substantially linear in the horizontaldirection.
 5. The method of claim 1, wherein the third image has morepixels than the first image.
 6. The method of claim 5, wherein some ofthe pixels of the third image correspond directly to pixels of the firstimage and wherein other ones of the pixels of the third image correspondto interpolated versions of some of the pixels of the first image. 7.The method of claim 6, further comprising interpolating said some of thepixels of the first image to produce said other ones of the pixels inthe third image.
 8. The method of claim 1, wherein the other one of thefirst and second images is a 2D camera image.
 9. The method of claim 8,wherein the first sensor comprises an array of photodiodes and whereinthe second sensor comprises a digital camera.
 10. The method of claim 1,wherein the second image comprises pixels that are distributed inaccordance with a substantially linear distribution function relative toa field angle of object points of the FoV.
 11. The method of claim 1,wherein the FoV comprises a vertical FoV and a horizontal FoV.
 12. Themethod of claim 11, wherein the vertical FoV spans at least 60 degreesand the horizontal FoV spans at least 150 degrees.
 13. The method ofclaim 11, the image point distribution function being non-linearrelative to a field angle of object points in at least the vertical FoV.14. The method of claim 11, the image point distribution function beingnon-linear relative to a field angle of object points in both thehorizontal FoV and the vertical FoV.
 15. The method of claim 1, whereinthe composite image is an RGBD image.
 16. The method of claim 1, whereinthe first image comprises at least one substantially compressed zone andat least one substantially expanded zone, and wherein correcting thefirst image comprises at least one of (i) compressing the substantiallyexpanded zone and (ii) expanding the substantially compressed zone, toproduce the third image.
 17. The method of claim 1, wherein capturingthe second image of the scene is carried out by sequentially capturingdifferent subportions of the FoV as illuminated by an optical signalemitted in a controllable direction.